System, apparatus and methods for adaptive data transport and optimization of application execution

ABSTRACT

Elements and processes used to enable the generation and interaction with complex networks, simulations, models, or environments. In some embodiments, this is accomplished by use of a client-server architecture that implements processes to improve data transport efficiency (and hence network usage), reduce latency experienced by users, and optimize the performance of the network, simulation, model or environment with respect to multiple parameters.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.62/062,790, entitled “System and Method for Interactive VirtualEnvironment,” filed Oct. 10, 2014, which is incorporated herein byreference in its entirety (including the Appendix) for all purposes.

BACKGROUND

Complex systems or environments often require simulation or modeling asa way of determining costs, identifying potential sources of error,testing new design features, etc. In addition, forms of entertainmentsuch as gaming often require similar types of resources and flexibilityas that of complex system models. These resources include computationalpower, data storage, processes for efficient data transfer acrossnetworks, and processes to provide users with the ability to vary systemor environment configurations (e.g., to change boundary conditions orconstraints) in order to observe how the system or model changes.

Unfortunately, conventional systems or platforms for generating andenabling interactions with complex networks, simulations, models, orenvironments (including virtual environments, such as found in gamingexperiences) are implemented in ways that limit their scalability, theirability to develop and incorporate learned behaviors, and theireffectiveness for users as a result of latency problems when largeamounts of data or content is transferred across a network.

These are important factors; for some types of systems or userexperience, latency is a limiting factor. For example, virtual realityapplications typically require low-latency interactions between a userand the environment in order to provide a satisfactory end userexperience. However, if such an application is hosted on a remote serveror platform, data transfer rates are impacted by networkcharacteristics, resulting in a sub-optimal experience. For this reason,conventional virtual reality applications and complex simulations ormodels may not be able to be effectively hosted as a cloud-basedservice.

Conventional systems also typically fail to implement sufficientoptimizations to reduce or more effectively manage CPU and/or resourceconsumption within a web-based or cloud platform application. Thisresults in higher demands on platform and/or client devices andprocesses, and can lead to load balancing and other resource allocationproblems.

Further, current Content Distribution Networks (CDN) face significantbandwidth constraints as a result of using conventional methods tohandle larger volumes of data. This is partially the result of how datais stored and transferred; as a result, it is difficult to know whetherinformation being removed from a file to reduce bandwidth constraints isimportant or not to the end user experience. As streaming video andother content delivery becomes more widespread, these constraintsprevent the growth and adoption of business models based on thosetechnologies.

Conventional systems and approaches also suffer from deficiencies withregards to data loss during compression, as data compression typicallyresults in uniform data loss. This is undesirable, as lossy compressionmethods may not provide a satisfactory user experience on highdefinition displays. To minimize this problem, conventional systems mayopt to perform only limited compression that does not significantlydeteriorate the end user experience. However, this approach may notaddress the data transport issues that lead to experiencing latency whenusing the system.

Embodiments of the invention are directed toward solving these and otherproblems individually and collectively.

SUMMARY

The terms “invention,” “the invention,” “this invention” and “thepresent invention” as used herein are intended to refer broadly to allof the subject matter described in this document and to the claims.Statements containing these terms should be understood not to limit thesubject matter described herein or to limit the meaning or scope of theclaims. Embodiments of the invention covered by this patent are definedby the claims and not by this summary. This summary is a high-leveloverview of various aspects of the invention and introduces some of theconcepts that are further described in the Detailed Description sectionbelow. This summary is not intended to identify key, required, oressential features of the claimed subject matter, nor is it intended tobe used in isolation to determine the scope of the claimed subjectmatter. The subject matter should be understood by reference toappropriate portions of the entire specification of this patent, to anyor all drawings, and to each claim.

Embodiments of the inventive systems, apparatuses, and methods aredirected to elements and processes used to enable the generation andinteraction with complex networks, simulations, models, or environments.In some embodiments, this is accomplished by use of a client-serverarchitecture that implements processes to improve data transportefficiency (and hence network usage), reduce latency experienced byusers, and optimize the performance of the network, simulation, model orenvironment with respect to multiple parameters.

These features of the inventive systems, apparatuses, and methodsprovide ways to improve the user experience when participating orinteracting with an immersive virtual environment, such as a gamingenvironment. The features also assist in improving the operation oflarge scale network simulations or models (such as for transportation,communications, or computational networks) and their constituent nodes,by reducing latency and enabling the development of localized controland intelligence. Further, the features permit a more efficient way tooptimize system performance in order to more rapidly evolve and improveoperation of the system, model, etc.

In one embodiment, the invention is direct to a system to provide a dataprocessing service to a client device, where the system includes:

an optimization platform communicatively coupled to a source of the dataprocessing service and configured to perform one or more optimizationprocesses on a data processing application that is hosted at least inpart by the service, on data, or on both;

a first optimization process residing on the optimization platform andoperable to identify an association between a first subset of a set ofdata and a second subset of the set of data;

a second optimization process residing on the optimization platform andoperable to determine one or more executable instructions that are partof the data processing application and that may be executed by theclient device in response to a signal received from the client device;

a data transfer process configured to provide the client device with thesecond subset of the set of data in response to a signal received fromthe client device requesting the set of data; and

an application transfer process configured to provide the client devicewith the one or more executable instructions that are part of the dataprocessing application and that may be executed by the client device inresponse to the signal received from the client device.

In another embodiment, the invention is directed to a process to reducethe latency experienced by a client device interacting with a serverdevice, where the client and server devices are part of a common system,and where the process includes:

for a plurality of pairs of events occurring within the common system,determining a probability of occurrence between a first event of a pairand a second event of the pair;

constructing a representation of a network, the network includingmultiple nodes with each node representing an event occurring within thecommon system, wherein a transition between two nodes in the network isassociated with the probability of occurrence between the eventrepresented by a first node and the event represented by the secondnode;

for each of a plurality of possible events occurring within the commonsystem and based on a state of the common system and the constructedrepresentation, determining one or more likely states of the commonsystem that may result in response to each of the plurality of possibleevents;

providing each of the one or more likely states of the common systemthat may result in response to each of the plurality of possible eventsto the client device;

operating the client device to generate one of the plurality of possibleevents; and

based on the generated event, configuring the common system to be in thestate resulting from the response to the generated event.

In yet another embodiment, the invention is directed to a process forreducing an amount of data that needs to be transferred over a network,where the process includes:

performing an optimization process that includes an associative neuralnetwork, the neural network including elements operable to determinethat a first event and a second event are generated within a specifiedtime interval;

using the associative neural network to identify an association betweena first set of data elements and a second set of data elements; and

providing the first set of data elements to the network.

Other objects and advantages of the present invention will be apparentto one of ordinary skill in the art upon review of the detaileddescription of the present invention and the included figures.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent application publication with colordrawings(s) will be provided by the Office upon request and payment ofthe necessary fee.

Embodiments of the invention in accordance with the present disclosurewill be described with reference to the drawings, in which:

FIG. 1 is a diagram illustrating aspects of one example of a computingnetwork architecture in which an embodiment of the invention may beimplemented;

FIG. 2 is a diagram illustrating elements of one example of anenvironment in which an embodiment of the invention may be implemented;

FIG. 3(a) is a flow chart or flow diagram illustrating a method,process, operation, or function that may be used to optimize theprocessing of events in an implementation of an embodiment of theinventive system;

FIG. 3(b) is a flow chart or flow diagram illustrating a method,process, operation, or function that may be used to optimize theprocessing of a software program or sequence of executable instructionsin an implementation of an embodiment of the inventive system;

FIG. 3(c) is an example of a Bayesian network representing the “flow” oflogic within a software application or set of executable instructions,and that may be used in implementing an embodiment of the inventivesystem and methods;

FIG. 4 is a state diagram illustrating an example of a Bayesianpredictive tree that may be generated for a set of events and that maybe used in implementing an embodiment of the inventive system andmethods;

FIG. 5 is a diagram illustrating a neural network that may be used toreduce the amount of data that is required to be transferred over anetwork in an implementation of an embodiment of the inventive systemand methods;

FIG. 6 is a diagram illustrating a “distance” metric that may be used todescribe the allocation of nodes within the neural network of FIG. 5;

FIG. 7 is a diagram illustrating how an embodiment of the inventivesystem and methods may be used to reduce the amount of data/informationthat must be sent over a network in order to permit a client sideprocess to reconstruct a larger set of data/information in a manner thatproduces a satisfactory user experience;

FIG. 8 is another diagram illustrating how an embodiment of theinventive system and methods may be used to reduce the amount ofdata/information that must be sent over a network in order to permit aclient side process to reconstruct a larger set of data/information in amanner that produces a satisfactory user experience;

FIG. 9 is yet another diagram illustrating how an embodiment of theinventive system and methods may be used to reduce the amount ofdata/information that must be sent over a network in order to permit aclient side process to reconstruct a larger set of data/information in amanner that produces a satisfactory user experience;

FIG. 10 is a diagram illustrating the process flow for implementing anembodiment of the inventive system and methods which reduces the amountof data that is required to be transferred over a network based onrecognizing certain common data sequences or “memes”;

FIG. 11 is a diagram showing certain elements or processes of an exampleadaptor device 1100 for providing an interface between an external userdevice 1102 and a data processing server or platform executing one ormore of the inventive optimization methods;

FIG. 12 is a diagram illustrating how a data processing server orplatform may be integrated with an embodiment of the inventiveoptimization methods (as implemented in an adaptor) in order to providea user with an improved experience when transferring relatively largeamounts of data over a network;

FIG. 13 is a diagram illustrating elements that may be present in acomputer device and/or system 1300 configured to implement a methodand/or process in accordance with some embodiments of the presentinvention;

FIGS. 14-16 are diagrams illustrating the operation of a neural networkthat may be used in an implementation of an embodiment of the inventivesystem and methods to recognize and utilize associations between eventsor data; and

FIG. 17 is a diagram illustrating how the identification of primitivesin an image may be used to reconstruct the image using less data.

FIG. 18 is a Table illustrating an example of multiple events occurringsimultaneously (in the example shown in the figure, there are tenindependent threads functioning, with each thread representing aspecific calculation) and how the application of a statistical analysismay permit the detection or observance of certain patterns and used todetermine a likely set of transformations that a VEnv might store inmemory.

Note that the same numbers are used throughout the disclosure andfigures to reference like components and features.

DETAILED DESCRIPTION

The subject matter of embodiments of the present invention is describedhere with specificity to meet statutory requirements, but thisdescription is not necessarily intended to limit the scope of theclaims. The claimed subject matter may be embodied in other ways, mayinclude different elements or steps, and may be used in conjunction withother existing or future technologies. This description should not beinterpreted as implying any particular order or arrangement among orbetween various steps or elements except when the order of individualsteps or arrangement of elements is explicitly described.

Embodiments of the invention will be described more fully hereinafterwith reference to the accompanying drawings, which form a part hereof,and which show, by way of illustration, exemplary embodiments by whichthe invention may be practiced. This invention may, however, be embodiedin many different forms and should not be construed as limited to theembodiments set forth herein; rather, these embodiments are provided sothat this disclosure will satisfy the statutory requirements and conveythe scope of the invention to those skilled in the art.

Among other things, the present invention may be embodied in whole or inpart as a system (such as a client-server or client-platformarchitecture), as one or more methods, or as one or more devices.Embodiments of the invention may take the form of a hardware implementedembodiment, a software implemented embodiment, or an embodimentcombining software and hardware aspects. For example, in someembodiments, one or more of the operations, functions, processes, ormethods described herein may be implemented by one or more suitableprocessing elements (such as a processor, microprocessor, CPU,controller, etc.) that are part of a client device, server, networkelement, or other form of computing or data processing device/platformand that are programmed with a set of executable instructions (e.g.,software instructions), where the instructions may be stored in asuitable data storage element. In some embodiments, one or more of theoperations, functions, processes, or methods described herein may beimplemented by a specialized form of hardware, such as a programmablegate array, application specific integrated circuit (ASIC), or the like.The following detailed description is, therefore, not to be taken in alimiting sense.

Disclosed herein are inventive systems and methods that improve theperformance of systems and devices in which they are implemented or withwhich they are utilized. These improvements may include reducing thelatency perceived by a user when interacting with a softwareapplication, data processing platform, or data transfer network. In someembodiments, the inventive system and methods provide for more efficienthandling of large amounts of data that are processed on a platformand/or transferred over a network (thereby reducing the demand onavailable computing and data transfer resources). In some embodiments,the inventive system and methods may be used to optimize the order ofexecution of complex software applications to provide a moresatisfactory end user experience. In some embodiments, the inventivesystem and methods represent a sequence of events or the logic flow ofan application as an operator acting on an initial state of a system toproduce the next state. In this representation, the softwareinstructions that are executed are viewed as “events” that have adefined flow based on the state of a system being controlled or impactedby the execution of the instructions.

Based on such a representation of events or program flow, the inventivemethods are able to “predict” or anticipate the next likely event orevents that will be generated or set of instructions that will beexecuted, and in response to pre-compute one or more expected nextstates that result from the events or execution of the instructions.These one or more expected next states are then stored. When the systemdetects the next event or actual instruction that is executed, it thenretrieves the appropriate next state (which may be an event or theresult of executing one or more application instructions). In this way,the user's perceived latency may be reduced, while also reducing thetype and amount of data (such as application modules or service relatedlogic/instructions) that might otherwise need to be transferred over anetwork.

In some embodiments, the inventive methods are also (or instead) able toidentify associations or relationships between portions of a set ofdata. This may be accomplished by use of an associative neural networkthat can “discover” an association between one subset of data andanother subset of the data. When found, such associations may be used tomore efficiently transfer sufficient information to reconstruct theentire data set over a network by transferring only a subset, andrelying on the knowledge of the recipient regarding the relationshipbetween the transferred subset and the not transferred data toreconstruct the entire data set. This can be an effective way to reducebandwidth requirements for the transfer of data that exhibits certainsymmetries, follows certain rules, etc.

FIG. 1 is a diagram illustrating aspects of one example of a computingnetwork architecture in which an embodiment of the invention may beimplemented. As shown, a variety of clients 130 incorporating and/orincorporated into a variety of computing devices 132, 134, 136 and 138may communicate with an “event prediction” or optimization system orplatform 150 through one or more networks 140. For example, a client mayincorporate and/or be incorporated into a client application (e.g.,software) implemented at least in part by one or more of the computingdevices. Examples of suitable computing devices include game consoles,personal computers, server computers, desktop computers, laptopcomputers, notebook computers, personal digital assistants (PDAs), smartphones, cell phones, and consumer electronic devices incorporating oneor more computing device components such as one or more processors,central processing units (CPU), or controllers. Examples of suitablenetworks 140 include networks utilizing wired and wireless communicationtechnologies and networks operating in accordance with any suitablenetworking and/or communication protocol (e.g., the Internet). One ofordinary skill in the art will recognize that other computing devicesand communication networks may be used to implement the invention.

In this example architecture, the Optimization Platform 150 includes anaccount/user interface 152. This interface may be operable to provideregistration and user authentication functions, to receivecommunications or data from a client side software application, and tocontrol the accessing of event data and/or application logic data for auser or aggregate of users, and the processing/evaluation of that data.In general, interface 152 communicates with client devices 130 vianetwork 140 and serves as a front-end to one or more functional modules(e.g., 156, 158, 160, etc.), the functionality of each of which may beimplemented by a programmed processor (e.g., CPU, micro-processor,controller, etc.) executing a suitable set of software instructions. Aswill be described in greater detail, Optimization Platform 150 may beimplemented in the form of one or more servers or other data processingelements, and may include functional modules or processes that areresponsible for one or more of the following functions, operations, orprocesses:

-   -   Account/User Interface (module 152)—responsible for registering        users or groups of users, verifying the authentication data        provided by a user, accepting and/or interpreting instructions        or commands from users or from user devices, and providing user        access to the data and/or functionality of optimization platform        150;    -   Event data store (module 154)—a data storage element for data or        identification of an event or sequence of events, typically        indexed by user and/or account;    -   Application Analysis (module 156)—a process or set of processes        that may be used to evaluate and analyze the flow of a software        application or program, or one or a sequence of executable        instructions—as described in greater detail with reference to        FIGS. 3(b) and 3(c), this module is operable to analyze the flow        of an application or program in terms of how the execution of an        instruction or set of instructions may affect the “state” of a        system that is being controlled or impacted by the application        or program;    -   Conditional Probabilities Model(s) (Bayesian Networks) (module        158)—this function or process is responsible for considering        data from event data store 154 and/or application analysis 156        to calculate the conditional probabilities between pairs of        events, executable portions of an application, or pairs of        states of a system that is being controlled or impacted by the        application or program—from these conditional probabilities, a        Bayesian network or networks may be constructed (as described in        greater detail with reference to FIGS. 3(a), 3(b), 3(c), and 4);    -   Pre-Computation/Determination of probable resulting event(s),        executed application instructions, or system state as a result        of application or program execution (module 160)—this process or        set of processes is operable to compute, determine, or otherwise        construct the result of detecting a specific event or sequence        of events, or the execution of an instruction or set of        instructions contained in the application/program, where the        specific event or instruction executed is one of those most        likely to occur, based on analysis of the constructed Bayesian        network(s);    -   Data store for pre-computed/determined result or end-state        (module 162)—this is used to store the results of the        pre-computation/determination stage, typically as indexed by the        user or account; and    -   Data Transfer Optimization Processing (module 159)—this is used        to determine a subset of a data set that may be used to        reconstruct the entire data set (or a satisfactory        representation of the full data set)—in some embodiments, this        may take the form of an associative neural network (an example        of which is described with reference to FIGS. 14-16) that is        capable of identifying/recognizing similarities, symmetries, or        other forms of relationships between portions of a data set        (such as an image, video frame, etc.) and using that information        to reduce the amount of data that needs to be transferred over a        network, while not compromising the user experience.

In some embodiments, the invention may be implemented within aclient-server architecture, such as a system consisting of a hardwareelement and/or a software client (as described with reference to clientdevice elements 130 in FIG. 1) and a server platform (as described withreference to the optimization platform, element 150 in FIG. 1). In someuse cases, the optimization platform may be used to calculate/implementoptimizations to improve the end user experience of participating in animmersive virtual environment or activity (such as a simulation, game,performance of a task, interaction with the development of a system overtime, etc.), by one or more of:

-   -   1. Reducing the volume of data needed to deliver a high quality        media/3D experience for a user or group of users;    -   2. Decreasing data transfer times over a network between a        system platform and one or more client devices;    -   3. Optimizing the execution performance of online media        playback, data transfer, and the execution of relatively complex        applications/programs;    -   4. Providing design and usage patterns for use by software        developers to improve their development of software        applications; and    -   5. Reducing the amount of un-needed data reaching an end user        (which can otherwise utilize resources unnecessarily, negatively        impact performance, and reduce user satisfaction).

In some embodiments, one or more components or processes of the systemmay achieve these results by learning usage/system evolutionpatterns—either those of a single user or in aggregate across multipleusers and/or applications, and then optimizing the performance of anapplication or another driver of a system (such as a state machine,applied operator, etc.). It is believed that no conventional technologycan perform the tasks performed by the inventive system concurrently, orat least not while achieving the performance enhancements of theinventive system and methods. One reason for this is because theinventive system operates to optimize both data throughput (data accessand transfer over a network) and application performance (asdemonstrated by a reduction in execution latency) at substantially thesame time.

The described program performance optimization process includes the useof predictive logic to determine (in advance of the actual need) a setof resources (or a configuration of them) that can be used to achievemaximum performance. This is valuable because in order for a program orsystem to interact efficiently with an end user, it is important to beable to anticipate user behavior and also to reduce the amount of data(such as content for an image, video, gaming environment, simulation,virtual environment, etc.) transferred over a network to the user, whilesimultaneously preserving a satisfactory experience for the user. Notethat one additional benefit of reducing the amount of data transferredis that it reduces the experienced/perceived latency even further. Asrecognized by the inventor, in order to provide the user with the bestexperience, the following functions need to be executed substantiallysimultaneously:

-   -   1. The application, service, or program is configured to provide        the needed resources at the right moment in time; to accomplish        this, the inventive system and methods compute multiple paths so        that when the user takes an action, the required end states or        results have been pre-computed and are available (this is        particularly valuable where a program or environment may have        several possible evolutionary paths); and    -   2. The actual latency between client and server should be        reduced; in embodiments of the inventive system and methods,        this is done by transforming the data into a form that reduces        the volume of data transmitted (without the user experience        being negatively impacted) and/or by sending partial information        and enabling the recipient to reconstruct the data based on        experience or prior knowledge.        Note that the various optimization processes, operations,        functions, or methods described herein may be implemented in any        suitable form, including but not limited to hardware, firmware,        software instructions that are executed by a suitably programmed        processing element, programmable elements (e.g., a        field-programmable-gate-array), etc. Further, these        implementations may be contained in or executed in part or in        whole by a server, platform, network of servers, client device,        etc., with aspects or portions of the processes, operations,        functions, or methods being performed on a server, on a client,        or shared between a server and a client.

In some embodiments, the inventive system, elements, and processes arebased on a client-server architecture (as illustrated in FIG. 1),wherein the server platform/system processes and handles optimizationseither on a per user basis, per application/program basis, or globallyacross an account or an aggregation of users. In the event that some orall of the inventive logic is embedded on a chip, the resultingarchitecture may be one such as that shown in FIG. 2. In this example,the client device driver 202 is used as an interface between the clientdevice 204 and an embodiment of the inventive adaptor/processor 206 (orother suitable element, such as a state machine, driver, ordelay-locked-loop) that is executing (or causing the execution of) a setof instructions or otherwise causing the evolution of a system, in orderto implement one or more of the inventive functions, operations, orprocesses.

In this example embodiment, processor/adaptor 206 includes a protocollayer 207 and a logic layer 208, which enable the processor to implementthe functions or capabilities used to provide the inventive services forthe client device(s). The output(s) of the inventive services may beprovided to a suitable output device (such as a display or viewer) 210via the client device driver 202. The client side elements and processescommunicate and exchange data with the elements, processes, and servicesresident on the server side of the architecture 212 using a suitablecommunications protocol 214.

The functions, operations, or processes that may be implemented as partof an embodiment of the inventive system may broadly be described asimplementing two classes/types of optimizations; (a) application/programexecution optimizations and (b) data access and data transferoptimizations. It is noted that both of these types of optimizations mayassist in providing more satisfactory performance for a user by reducingperceived latency and more efficiently using system resources, includingthe utilization of a processor and a data transfer and/or communicationsnetwork. This may be helpful in situations in which a user is accessingthe services of one or more applications that are resident on a remotedata processing platform and/or where the user is accessing orinstructing an application to access data stored on the platform (suchas business related data, media content, etc.).

As part of the inventive approach to accelerating performance within asystem that communicates/transfers data using events and/or executes aresource intensive program, embodiments of the inventive system mayimplement a form of Bayesian caching of object behavior. In one usecase, based on accumulated event data from the experience of a largenumbers of users and objects, Bayesian predictive caching may beutilized to improve the timeliness and responsiveness of a system toevents. In addition, as will be described, the principles behind theinventive approach to event caching may be applied to the optimizationof the execution of a software program or instruction set that operateson a state of a system to generate a next state.

In terms of event modeling, Bayesian or predictive caching is based onthe ability to make “predictions” as to what data will be needed or whatevent will occur (or is likely to occur) next and to (pre)generate andstore the data, system state, or values that are most likely to beneeded or used in response to those event(s) occurring. In someembodiments, a system may use this technique to reduce the flow ofnetwork traffic (thereby reducing the bandwidth needed) and/or theoverhead associated with decrypting memory content by making predictionsas to what would be the most likely next state or states. In the case ofsome embodiments of the inventive system and methods, predictive cachingmay be applied to events occurring within a system that are typicallyemitted or received by an entity. Note that from one perspective, anevent occurring within a system will generally result in an action on,or transformation of the state of the system.

When an event occurs, it triggers an operation, transformation, ortransition between system or environment states. If a reasonablyaccurate prediction can be made as to which event or events will takeplace next, then it is possible to pre-calculate the resultingtransformation/transition and store it in memory in anticipation of theoccurrence of the predicted event or events. This may be used to improvethe response of the system and deliver a more seamless experience to auser. For example, suppose that Event 1 happens and this causes anaction A and then, shortly thereafter, Event 2 happens, which triggersaction B. Statistically, the system determines that Event 2 isrelatively highly likely to occur after Event 1, which means that ifEvent 1 occurs, transformation B may be calculated and the resultantstate stored in memory until it is needed. In such an example, theresultant state may be presented to a user more quickly than in thesituation where such a “pre-computation” does not occur.

In general, there are two types of “events” that are considered by anembodiment of the inventive system and methods. The first aretraditional application/program generated events. In the case ofevaluating these events, an application may be instrumented along keydecision branches and the events originating from these branchesanalyzed. These events are those with a branch identifier (most likelyan instruction offset) and a timestamp. The identifier is a 64 bitvalue, the time sequence is a 64 bit value, and the instruction offsetis a 64 bit value. Such an event is a 128 bit value that is written to aspecific set of virtual registers every time a decision has been made ona given branch. These events are collected asynchronously andreconstructed into a Bayesian network (as described with reference toFIGS. 3(b), 3(c), and 4). In this example, as the client executes partsof an application, program or process, the likely alternative logicflows are loaded on the server side so that they may be delivered to theclient in time for execution. Using this approach, the entire programdoes not need to be sent to the client—instead, only the portions orinstruction sequences that the program needs to execute based on theevolution of the program logic or the system being controlled by theprogram are provided to the client.

Similarly, in a system that includes entities that exchange informationor data using messages or another form of emitted and received events,embodiments of the inventive system and methods may be used to constructa Bayesian network of possible messages/events and their impact on thestate of a system in which those events or messages are being exchanged.In this case, when an event of a certain type is detected, the inventivemethods may determine the next likely set of events and pre-calculatethe system state as a consequence of each of those events. Thisembodiment is described with reference to FIGS. 3(a) and 4.

Note that for either type of “event” and its associated networkrepresentation, the inventive system may further optimize aspects of theperformance of the controlled system or environment by determining how auser will provide inputs to control a program flow, and then use thatinformation to further reduce the set of likely/possible outcomes.

FIG. 3(a) is a flow chart or flow diagram illustrating a method,process, operation, or function that may be used to optimize theprocessing of events in an implementation of an embodiment of theinventive system. In this method, process, operation, or function, asequence of events is represented as an operator acting on an initialstate of system to cause it to move to a next state. Note that theinventive approach is able to optimize in a way that permits transfer ofa specific instruction or sequence of instructions, instead ofoptimization that is applied at the granularity level of a module orsub-routine.

As shown in the figure, events occurring within the system are monitoredor detected. These events may include only those involving a certainnode or active element, those involving a defined group of nodes (e.g.,those with a specified characteristic or parameter value, thoseassociated with a specified account, etc.), or every node, for example,as suggested by step or stage 302. The data is then processed todetermine one or more conditional probabilities between pairs of events,as suggested by step or stage 304. Based on the determined conditionalprobabilities, a Bayesian network of connected nodes is constructed,where each node represents a state of the system, as suggested by stepor stage 306 (an example of which is described in further detail withreference to FIG. 4).

Next, an event is detected, where that event was originated by a user orby another node, as suggested by step or stage 308. Using the Bayesiannetwork model, the conditional probabilities that were determined areused to “predict” the next likely event or events, as suggested by stepor stage 310. For each “predicted” event the outcome of that event isgenerated and stored, as suggested by step or stage 312. Note that insome embodiments, this means that the state of the system after exposureto each of the likely or predicted events is determined and stored.Next, an actual event or operation that occurs is detected and/ordetermined, as suggested by step or stage 314. Based on the actualevent, the appropriate previously computed/determined system state isaccessed and presented to the user, as suggested by step or stage 316.

In some embodiments (as described with reference to FIGS. 3(b) and3(c)), when an application (i.e., a set or sequence of instructions) isloaded into the inventive system, it is processed by performing ananalysis which evaluates the program in terms of instruction sets (e.g.,Assembly). This may include placing event markers in the code aroundcertain key instruction blocks. The program is then traced through asimulator that identifies possible execution several paths (note thatthe set of paths does not need to be complete). Using the executionpaths/event streams, a network of “probable” event paths and systemstates is constructed. As a result, when a user executes the applicationcode, the system has information regarding what the next steps in theprogram are likely to be, and the system is able to pre-load theinstructions and pre-execute those instructions, storing the resultingstate(s) in memory. This builds a precognition model for the applicationprogram.

FIG. 3(b) is a flow chart or flow diagram illustrating a method,process, operation, or function that may be used to optimize theexecution of a set of instructions or a software program in animplementation of an embodiment of the inventive system. In this method,process, operation, or function, the result of executing one or moresoftware instructions is represented as an operator acting on an initialstate of system to cause it to move to a next state.

Initially, a software program, application, module, or sequence ofinstructions is evaluated to determine information about the manner inwhich a system under control of the program transitions from one stateor set of conditions to another. In this evaluation process, the “logic”of the program or application is examined to identify branch points,decision paths, and the general “flow” of the program that results fromthe execution of an instruction or a sequence of instructions (such asthose contained in a module or sub-routine). In some ways this is a formof tracing the logic flow of a set of instructions to understandbranching behavior and how the execution of an instruction or sequenceof instructions may be followed by a next instruction or sequence ofinstructions. This allows the inventive system and methods to develop a“model” for how control may flow through a program based on the state ofthe program (or of a system it is controlling or impacting) and anyrelevant events that occur at a given point in the execution of theprogram.

As shown in the figure, events (such as nodes in the execution/logicflow of a program) occurring within the system/program are monitored ordetected. If monitored, these events may include only those involving acertain node in the program logic, those involving a defined group ofnodes (e.g., those with a specified characteristic or parameter value,etc.), or every node, for example, as suggested by step or stage 322.The data and/or the logic flow model developed for the program is thenprocessed to determine one or more conditional probabilities betweenpairs of events (such as between states of a system controlled by aprogram or between execution of one or more instructions and a secondset of instructions), as suggested by step or stage 324. Based on thedetermined conditional probabilities, a Bayesian network of connectednodes is constructed, where each node represents a state of the system,as suggested by step or stage 326 (an example of which is described infurther detail with reference to FIGS. 3(c) and 4).

Next, an event, execution of an instruction, etc. is detected, wherethat event was originated by a user or by a node in the program flow, assuggested by step or stage 328. Using the Bayesian network model, theconditional probabilities that were determined are used to “predict” thenext likely event or events (e.g., portion of the logic flow, etc.), assuggested by step or stage 330. For each “predicted” event the outcomeof that event is generated and stored, as suggested by step or stage332. Note that this means that the state of the system after exposure toeach of the likely or predicted events (executed instructions, etc.) isdetermined and stored. Next, the actual event or operation that occursis detected and/or determined, as suggested by step or stage 334. Based,on the actual event, the previously computed/determined system state isaccessed and presented to the user, as suggested by step or stage 336.

FIG. 3(c) is an example of a Bayesian network representing the “flow” oflogic within a software application or set of executable instructions,and that may be used in implementing an embodiment of the inventivesystem and methods. As shown in the figure, the state of the program (orof a system controlled or impacted by the program), may transition fromone state (e.g., “A”) to another (e.g., “C”) at a certain time or pointin the execution. This will occur with a likelihood based on certainprobabilities that can be determined (where the probabilities may dependon the initial state and other inputs to the system). By understandingthe probabilities of transitions between different states of theprogram, one can construct a simple Markov chain. This enables thesystem to load multiple possible/likely pathways into memory based oncalculated probabilities and when the user takes a given action, thesystem has already (pre)computed the results.

FIG. 4 is a state diagram illustrating one example of a Bayesianpredictive tree that may be generated for a set of events, as describedwith reference to FIG. 3(a). In this example, assume that based on pastbehavior data, it is observed that if event A happens at T1, then it ishighly likely that transformations B and C will occur at T2 (where Band/or C may represent operations, decisions, etc.). In such a case, animplementation of a system using this Bayesian tree may loadtransformations B and C into memory so that the system can more quicklyprovide B or C if it occurs at T2. Further, assume that the pastbehavioral data also show that if events B and C occur then it's likelythat A will occur again at T3 and also highly likely that B will occuragain at T4. Further, events E and D each have a relatively lowprobability of occurring. Therefore, the system stores events A, B and Cin memory because it is likely that these events will occur again. Thus,the figure represents a Bayesian network illustrating conditionalprobabilities that might arise from a sequence of Events within asystem.

As illustrated in FIG. 18, in some cases multiple events may occursimultaneously. In the example shown, there are ten independent threadsfunctioning (with each thread representing a specific calculation—forexample, components of a climate model) and each of them are capable ofreceiving events. From the application of a statistical analysis,certain patterns may be observed and used to determine a likely set oftransformations that a system might store in memory. In the examplebelow, the Bayesian network indicates that if the transformation set{H,M,C,G,F,A} is loaded into memory, then for over 60% of the time, thesystem will not need to make a network query or decrypt data even thoughthe set represents only 30% of the possible transformations (theseobservations are based on a counting of events/transformations within agiven time interval ΔT).

Note that typically, Bayesian networks are represented by a-cyclical(that is, non-cyclical) graphs. However, in the context of a Markovchain that is time distributed, a cyclical Bayesian network may be moreeffective. The reason for this is that A at time T1 is not the sameobject as A at time T2. So even though the graph appears cyclical, ifthe network is ordered by time, then it is a-cyclical.

This behavior results because normal (typical) Bayesian networks don'tdifferentiate between something that occurs at time T1 and the sameevent occurring at time T2. For example, a medical expert system doesn'tcare about time in this regard and so an event A (person has a fever) attime T1 is the same as event A (the same person has a fever) occurringat T2. This is largely because time is not a relevant dimension in thissituation. However, in the inventive system (and the applications or usecases in which it is implemented), an event A occurring at T1 isconsidered to be different from the same event A occurring at T2 (i.e.,time is a differentiator).

Another example of an optimization process that may be applied using anembodiment of the inventive system and methods is one based on thedetection of saliency (i.e., a salient feature, aspect, event, etc.).Saliency is the property of a system to recognize when something isdifferent and hence may be indicative of an important feature or event.Note that being able to detect differences is not always simple orstraight forward. For example, a child can pick out a red ball from abunch of black balls. However, a computer may not be able to do thisefficiently unless the computer understands the concepts of what is redand what is black, and that red is the uncommon item in the set.

To address this problem, the inventors have constructed a neural networkthat exhibits the property of saliency, i.e., it is capable ofrecognizing a salient event or aspect of a system. This inventive neuralnetwork is described with reference to FIGS. 14-16, which are diagramsillustrating the operation of a neural network that may be used in animplementation of an embodiment of the inventive system and methods torecognize and utilize associations between events or data. The inventivenetwork includes an associational engine that “learns” from a system bymapping out “event” patterns based on inputs. In this sense, the networkand its nodes don't have to be pre-programmed but are capable oflearning on its own. If the neural network detects/observes somethingthat does not match the patterns it has previously encountered andmapped, then it identifies the event as “different”. Note that adifferent or uncommon event or aspect may be indicative of something ofimportance, of something suggesting an error, or of something that israre but not of importance.

The inventive saliency identifying neural network is capable ofdetecting events within multimodal systems. For example, it's possiblethat a system can exist in multiple modes; in this case, the neuralnetwork can observe the development or evolution of the system and learnthat the system has multiple modes because the network forms aprobability mapping of the system that shows connected clusters inn-dimensional space. If the space of modes that a system can occupy isnot a set of disjoint spaces but has transition paths between thespaces, then it is considered the normal behavior of the system. If adisjoint space appears, then it is inferred that the system hasundergone a change. For example, suppose a system includes a set ofblack balls and black cubes, and one red cube. Then the set of balls andcubes, although they are different, form a non-disjoint set; theconnecting property is the fact that they are black and they havecounts>1 i.e. {black, cube, count>1}, {black, ball, count>1}. On theother hand the red cube exists at a point {red, cube, count=1}, whichmeans that it is the most different as it is the least connected of the3 sets (the others have 2 overlapping dimensions—black and count>1).

This observation is useful because the associational aspects of theneural network of FIGS. 14-16 is capable of identifying the nominalbehavior of a system on its own (i.e., it does not need to be trained).For example, by understanding what an application's nominal behavior is,a system that is executing the application can pre-load data andresources to more efficiently support that behavior. If the nominalbehavior changes, then the system is able to adapt. For example, if aprogram, service, or application has been behaving in a certain way as aresult of user action and the user changes his or her behavior, then theprogram detects this change and modifies its operation to the newbehavior. If the system did not behave in this way, then once a userchanged his/her behavior any adaptation to the new user behavior mighttake a relatively long time (or the user would have to reset/reconfigurethe program to learn the new behavior).

However, if an event or system state deviates from the nominal behavior,then the system can determine that, for example, the user's usagepattern has changed, an error is about to happen, or unexpected data hasentered the system. This ability to recognize salient oruncommon/deviating events or states can be used to more efficientlyallocate resources. As mentioned, if a system understands nominalbehavior, then it can allocate resources accordingly. If, on the otherhand, it detects a fundamental shift in behavior, then it can alert aserver to allocate more resources (e.g., computational resources, datastorage, error detection, back-up data processing, etc.) as somethingunexpected is about to happen.

With regards to FIGS. 14-16, over time, associations in the network formas events occur simultaneously or near simultaneously. Recognizing thisnear simultaneity (and hence a possibly relevant “association”) isenabled by each node having memory of a signal being fired; this isaccomplished by retaining an active or “signaled” state for a shortperiod of time, ΔT. The pathways between nodes are strengthened as aresult of a coordinated set of events being fired. The network is then“trained” automatically (as illustrated in FIG. 14, showing theformation of an associative pattern) and a set of recall pathways(elements 1402, the dashed lines) are formed.

Once the network has learned one or more associations between events,then when a subset of events in this pattern of events is fired, thenetwork is able to recall the other events that are likely to be firedas a result of a feedback process (as illustrated in FIG. 15, showing anexample of the Recall function); in response, the network and/or systemwill then allocate the resources necessary for efficient operation.

However, if the “expected” pattern changes, then new pathways(corresponding in some cases to different event or logic flow) arenaturally formed (as illustrated in FIG. 16, showing how newassociations are formed); this occurs without the user having toreset/retrain the system. Note that in one embodiment, the feedforwardsignals follow the nearest neighbor stochastic method using a discreteGaussian distribution that is in accordance with the distributiondescribed with reference to FIG. 6.

Another class of use cases of embodiments of the inventive system andmethods (in addition to that of event sequencing or instructionexecution optimization, which typically involve pre-determining alimited set of the next possible events or system states, and then(pre)computing the impact of those events or changes in system state) isthat of determining the type and amount of data to transfer over anetwork in order to conserve bandwidth and decrease perceived latency,while providing an acceptable user experience or end result. This can beparticularly important for certain types of applications such as gaming,interaction with virtual environments, distributed computing,client-server architectures that exchange data over a channel withlimited bandwidth or bandwidth constraints, network models in whichcommunications between nodes are impacted by channel limitations orchannel variances, certain types of content delivery (e.g., real-timestreaming of video), etc.

Embodiments of the inventive system and methods can address the need fordata optimization by performing these operations or functions in one ormore of several ways, depending upon the type of data being exchanged.This dependence upon the type of data is necessary because some types ofdata are not as easily amenable to compression without a significantloss of fidelity, for example, 3-dimensional meshes. For instance, atypical mesh can contain over 20,000 polygons. To be most effective forthis type of data, a compression process needs to be applied at 3different levels; the geometry, the topology (which describes thecurvature of the mesh), and the attributes of the mesh (such as thecolor, texture, etc.). This necessitates dealing with a large amount ofdata, where the fidelity obtained after compression may depend on thecategory of mesh data being processed.

In some cases, an embodiment of the invention may use a mesh compressionmechanism that represents an application of discrete differentialgeometry to determine a measure of the topology (i.e., the localcurvature) of a mesh on the server side. This information is transferredto the client and then used to efficiently reconstruct certaingeometrical aspects of the mesh on the client side. In this embodiment,the local curvature of a space or figure (such as a mesh used to definea 3D element) is represented by less data than would be needed torepresent a portion of the space or figure in its entirety. This reducedform is transferred and then used to reconstruct the region or elementon the client side based on client-side knowledge regarding therelationship between the reduced format representation and thereconstructed portion or portions. This can be used to reduce the amountof data needed to be transferred over a network while retaining theability to utilize complex shapes as part of a simulation or virtualenvironment.

Embodiments of the inventive system and methods may also make use ofother techniques to reduce the amount of data needing to be transferredover a network, while retaining a satisfactory user experience. In oneembodiment, a specific type of neural network is constructed andutilized; an example of this neural network is described with referenceto FIG. 5 (which is a diagram illustrating a neural network that may beused to reduce the amount of data that is required to be transferredover a network in an implementation of an embodiment of the inventivesystem and methods). The illustrated network 500 includes multiplelayers 502, a feedback mechanism 504, and is massively inter-connected.Such a neural network structure may exhibit one or more of the followingproperties:

-   -   A layer that enables fast recall;    -   Multiple recall loci;    -   A hierarchy that allows new “memories” to form; and    -   A feedback loop that gives the network the ability to form        connections between disparate inputs.

As recognized by the inventor, the neural network illustrated in FIG. 5provides several important capabilities. For example, by detecting(i.e., identifying or forming) associations between seemingly unrelatedaspects of a system, the network enables the reconstruction of data frompartial information. Further, by allowing new “memories” to form, thenetwork can continue to grow and learn new patterns (and thereby infernew associations or connections between initially disparate elements).In addition, the structure permits an efficient separation of the neuralnetwork into a client resident and a server resident portion. In someembodiments, the client resident portion would retain the recallinformation that is relevant to a specific user and which corresponds toa specific recall locus.

As mentioned, in some embodiments, the elements or layers of the neuralnetwork illustrated in FIG. 5 may be distributed between the client andthe server, where the network is able to learn over time. The clientside elements are responsible for obtaining or inferring informationabout individual users and then transferring that information to theserver platform, which is then able to construct a more complex networkover time.

In one example implementation, the neural network may operate in thefollowing manner. The network includes several functional layers (502).Layer 0 (506) is the perception or input layer. Signals from Layer 0 arepassed stochastically to layer 1 (507), from layer 1 to layer 2 (508),and from layer 2 to layer 3 (509). Now assume that a signal is generatedby an external sensor or device. When the signal (or its processedversion) passes to an n−1^(th) layer of the neural network, the n−1^(th)layer will pass it onto the n^(th) layer. This transmission of a signalis performed as a stochastic process; the signal will be passed to then^(th) layer after weighting by a Gaussian drop-off function that isdependent on the “distance” or network metric between the nodes in then^(th) layer and the transmitting node in the n−1^(th) layer. Thisdistance measure or network metric is described further with referenceto FIG. 6 (and may indicate the number of “hops” a signal takes, thenumber of nodes traversed, or a weighted measure of the connectionbetween two nodes, etc.).

FIG. 6 is a diagram illustrating a “distance” metric that may be used todescribe the allocation of nodes within the neural network of FIG. 5,and may be used in constructing and operating the neural network. In thefigure, γ_(n) represents the “distance” or value of a network metricbetween a node in layer n−1 and a node (or nodes) in layer n. In oneimplementation, the distances between a node (602) in layer n−1 andmultiple nodes (604) in layer n may be distributed according to adiscrete Gaussian distribution:

${P_{c}( {\gamma_{n},s} )} \approx {\exp ( {- \frac{\pi {{\gamma_{n} - c}}^{2}}{s^{2}}} )}$

Further, due to the application of this weighting, if a node receives asignal from a particular node, then it will suppress signals from nearbynodes. This means that “frequent” signaling using a given path willstrengthen that path relative to the other paths. Referring to FIG. 5,at the highest level, there is a stochastic connection made to adjacentnodes with the same Gaussian drop-off behavior that is used to form aconnection from the n−1^(th) layer to the n^(th) layer. There is also afeedback mechanism from the highest node to the lowest node so that“familiar” things can be associated with new “memories”. Note that theimplementation and operation of such a network is similar to thatdescribed with reference to FIGS. 14-16.

In one embodiment, the 0^(th) layer of the neural network (layer 506 inFIG. 5) is maintained on the client side; this enables faster recall.However, note that if the client doesn't understand or recognize anevent, then it will send the signal/data to the server based layer(s).On the server side, if the server has seen the signal/data from theclient previously, then other relevant data may be sent to the client.

If the server doesn't recognize the signal/data provided by the client,then it will form a new “memory”. It will also associate thatsignal/data with other inputs arriving simultaneously or within a timewindow T (i.e., within a specified ΔT, which may be a factor of one ormore network characteristics, user characteristics, server/platformcharacteristics, etc.). Note that this does not imply causality, butonly that certain events are typically occurring within the same timewindow. This association causes frequently seen data or frequentlyassociated events to have a relatively stronger link to each other. Thisresults in them being returned more quickly (and often together) thanmore loosely associated events.

This behavior can be beneficial because if associations between events,signals, or data have been established through multiple input channels,then in some embodiments, the associated events, signals, or data areprovided in advance to the client in anticipation that they will beneeded at some point. For example, if a previous signal has triggeredthe creation of a “memory”, then new data may be associated with thatmemory. If the association is sufficiently strong (i.e., frequent orreliable), then that memory is bound to the new data as an association.An example of this would be that if incomplete data is detected, then ifthe complete data has been seen before, the server will access andprovide the complete data. Based on these concepts, the followingscenarios may arise:

-   -   Incomplete data is received—the client has seen something like        it before and “remembers/recalls” what the complete data looks        like, and returns the complete data;    -   Incomplete data is received—the client has not seen anything        like it before. The client sends that data to the server. The        server has seen the data and will return the complete data, with        client “remembering” this situation for the future;    -   Incomplete data is received—both the client and server have not        seen anything like it before. In this case, the data is stored.        Over time, the server builds up the complete data set (assuming        that something like the data is seen again) and is then able to        recall the data from incomplete bits in other cases.

This type of data handling system can be used in several valuable ways.For example, a server can operate to parse a photo, image, or other formof media, and construct a partial data block that it sends to a client.If the client has seen something like it before (i.e., it is recognized,partially recognized, or seems close/familiar), then the client may beable to reconstruct the set of data in a satisfactory manner. If thepartial data block is not recognized (e.g., due to being new orinsufficiently similar), then the client will make a call to the serverand request the full data block. To improve the efficiency of thisprocess, the data block may be broken into elements that are most likelyto be seen again (e.g., letters, simple words, etc.). Examples of thiscapability to reconstruct a more complete set of data from a subset ofthat data will be described with reference to FIGS. 7-9.

FIG. 7 is a diagram illustrating how an embodiment of the inventivesystem and methods may be used to reduce the amount of data/informationthat must be sent over a network in order to permit a client sideprocess to reconstruct a larger set of data/information in a manner thatproduces a satisfactory user experience. As shown in the figure, in somecases the amount of data shown in image 702 (in this case, a partialimage of text) may be provided from the server to the client over asuitable network. Even though the image is partial/incomplete, theclient is able to construct the corresponding image shown in image 704,via application of the methods and processes described herein. Forexample, in this case the following process steps may be implemented topermit the client to be able to construct the full image from thepartial data that it receives:

-   -   server processes original complete image;    -   server determines associations between portions of image (based        on symmetry, similarity, tags, object types, learned        relationships obtained from evaluating a larger data set,        recognition of standard shapes, etc.);    -   server constructs data set that represents portion of full data        set (and if needed, information regarding certain learned        associations/symmetries), and sends partial data to client; and    -   client receives subset of data set, and by knowing the relevant        associations, reconstructs the image (at a level of fidelity        which was already determined to be satisfactory).    -   For example, a shape may be broken up into components, such as        edges, color maps, and a low resolution image (i.e., one where a        substantial number of pixels have been removed). Next, an        embodiment of the inventive associative network is trained based        on these components. Note that if the system recognizes enough        components, then it is able to recall the original with a fairly        high fidelity (and one that may be satisfactory for many        purposes).    -   To enable this process to scale effectively, the system may        remember “canonical forms”. It may process enough eyes to figure        out some basic canonical shapes. Later, if it is presented with        points that are similar to one of its canonical eyes, it is able        to reconstruct the eye and then “morph” the canonical eye into a        more exact shape given the available points.    -   The system “learns” about shape primitives and associations by        looking at a set of images. It doesn't know that the shape it is        considering is an eye, but instead picks up repeated patterns        and uses a median value algorithm to calculate the corresponding        canonical shape.    -   Note that there are almost always repeating shapes in a set of        pictures (polygons, for example). For video, a similar method        may be applied, except that a frame by frame smooth transform is        used with interpolated mid points. In this way the process can        skip multiple frames and still reconstruct an image        sufficiently. Each frame can further be decomposed using the        approach described previously with reference to FIG. 7.    -   One issue is that of how the system knows that two points are        actually the same point in different frames. This can be        determined by decomposing a first frame into known primitives        and then decomposing a second frame into known primitives (as        suggested by FIG. 17). It is very unlikely that two identical        primitives are in close proximity, and so the system is able to        map the two points to each other and thus calculate the        intermediate frame. Once the mid frame has been reconstructed        then it is completed by using the associative network that is        described herein. The experience of the end user is to receive a        stream; as the stream of data is greatly reduced, more of the        movie is buffered in a given time interval. Once the stream is        downloaded into the inventive adaptor, the adaptor reconstructs        the video.

FIG. 8 is another diagram illustrating how an embodiment of theinventive system and methods may be used to reduce the amount ofdata/information that must be sent over a network in order to permit aclient side process to reconstruct a larger set of data/information in amanner that produces a satisfactory user experience. As shown in thefigure, in some cases the amount of data shown in image 802 (in thiscase, a partial image of text) may be provided from the server to theclient over a suitable network. Even though the image ispartial/incomplete, the client is able to construct the correspondingimage shown in image 804, via application of the methods and processesdescribed herein. In this case the process steps described withreference to FIG. 7 may be implemented to permit the client to be ableto construct the full image from the partial data that it receives.

FIG. 9 is yet another diagram illustrating how an embodiment of theinventive system and methods may be used to reduce the amount ofdata/information that must be sent over a network in order to permit aclient side process to reconstruct a larger set of data/information in amanner that produces a satisfactory user experience. As shown, morecomplex reconstructions of an image might be implemented by keepingedge, pixel, and color primitives on the client; this would allow theclient to infer the complete image without having seen the originalimage.

One technique for this is to decompose the image instead of usingprimitives. Using a base smoothed color map taken by calculating theaverage of a set of points and a reduced image (obtained removing alarge number of pixels), as well as a set of edges that describe shapesin the image, the system smooths out areas that don't vary much based onthe edge map (i.e., areas that don't show many edges). Regions with arelatively large number of edges are kept more or less intact in thereduced image, as the system assumes that those areas are detailed andremoving pixels will be more noticeable. In areas with no edges, thesystem can remove a number of pixels and reconstruct the image based onthe “chunking” method described next. The color map and the reducedimage map allow the system to interpolate colors of pixels between knownpixels. As areas near edges are highly defined, this allows the systemto color adjust those points adjacent to the highly defined areas; theresulting color adjustment cascades through the rest of the image andresults in an image where the colors are very similar to the originalimage.

In some embodiments, data optimizations may be used that operate topartition or “chunk” data on the client side into data sequences or“memes”, and to store this information in a table that is shared withthe server. After such a table or list is shared with the server, whenthe server sends data to the client, it may first lookup the shared datameme table and replace one or more data memes in the data set that itwas planning to send to the client with the applicable address offsetsfrom the table. In one embodiment, the data memes may be determined oridentified probabilistically, such as by use of a Bayesian networkconstructed from a timer-ordered sequence of events (examples of whichhave been described herein). In this use, the client might periodicallyupdate the server as the meme table is updated or revised.

In such a situation, the process flow might appear as shown in FIG. 10.As a result of having a shared data table (in this case the data memetable), event message sizes can be decreased by replacing certain datasequences by a pointer to a data sequence stored in the data table, andspecifically sending data where such a pointer does not exist. Thus, inthe example shown in FIG. 10, the server will perform a lookup in theshared table (1002), replace the appropriate sequences with pointers tolocations in the table (1004), and then send the reduced sized event(consisting of the addresses of pointers to recognized memes and actualdata for new memes) to the client.

As shown or suggested in the figure, data may be analyzed on a clientand/or on a server. Common memes are identified based on building aprobability table. Memes that are relatively highly likely to arise arekept in a shared table with known offsets. When one of these likelymemes is encountered in a stream of data that is going to betransmitted, it is replaced by a pointer to the entry in the sharedtable. An example is that in the English language, some words are morelikely to occur such as “the”. Instead of sending the whole word, apointer to the location in the table is sent, and as the table is sharedbetween client and server, the original word is able to be re-inserted.The shared table may be refreshed periodically, as the most likely memesmay change.

In some embodiments, aspects of the inventive system and methods may beused to enable more efficient interaction of a user's device with asystem or platform that is hosting a simulation, model, or virtualenvironment, for example. This can be accomplished by use of an“adaptor” that operates to couple a user's data input device to a remoteserver or data processing platform, while providing mechanisms for theefficient exchange of data and messages between the user's device andthe platform over a network. A description of one or moreimplementations of such an adaptor will be described with reference toFIGS. 11 and 12.

Communications and data exchange between a client process and theprocesses on a server/platform may be accomplished using a client devicethat is optimized for high bandwidth communications, computationallyintensive operations, and portability. In some examples, aserver/platform may provide a universal I/O and computational layer thatenables distributed computation and I/O across multiple machinesspanning multiple physical locations. Examples of the types of devicesthat may be used as part of the inventive system include tablets, smartphones, desktop computers, laptops computers, gaming consoles and 3Dvisors. In some cases, the adaptor may be a separate hardware deviceconnected to a computing device, and in other cases the adaptor may be asoftware program or a component of a software program executing on acomputing device.

FIG. 11 is a diagram showing certain elements or processes of an exampleadaptor device 1100 for providing an interface between an external userdevice 1102 and a data processing server or platform executing one ormore of the inventive optimization methods. The illustrated adaptordevice utilizes multiple multicore floating point calculators 1104 toaccelerate the processing of signals between the external device 1102and the system platform. For example, signal routing may be implementedby the adaptor, wherein incoming packets on a Wi-Fi NIC 1106 aremonitored by a signal router 1108 that looks for a flag or signal bitthat causes data packets to be diverted to specialized hardwareconfigured for device signal encoding/decoding.

In the illustrated example, the adaptor 1100 includes an embeddedBayesian intelligent caching service 1110 (such as that described withreference to FIGS. 3(c) and 4, or other predictive pattern matchingbased process or service) for use in predictive processing and/oroptimization operations. The Bayesian network may be configured to beupdatable from the server/platform in order to obtain Bayesian networkdata from a larger (i.e., aggregate) dataset, such as one based on datafor multiple users or accounts. Predictive processing may be used toreduce perceived latency and improve the user experience, such as bydownloading partial images or data sets in advance of a time when theywill be needed based on predictions of the data that might be used(e.g., based on past activity for a game or for a user).

Note that predictive data/resource allocation may utilize considerationsof saliency to allocate additional resources for processing signalactivity. For example, if activity in a system deviates from establishedpatterns (suggesting that it might be more relevant in that situation),then an unsupervised learning system can operate to recognize theactivity/deviation and generate requests for additional resources toprocess the new activity.

In contrast, conventional systems typically reside on a local device andutilize an assumed Poisson distribution for generating predictions.Other conventional implementations may be reactionary and allocateresources in response to a spike in usage instead of anticipating theincreased need for resources; this approach can result in greaterlatency. However, the predictive approaches used in some embodiments ofthe inventive system and methods may utilize self-learning forintelligent resource allocation in order to increase the performance ofa cloud architecture for hosting a virtual environment, as well assupport scalability and more efficient use of resources.

Referring to FIG. 11, an embodiment of the inventive adaptor may includea network adaptor and a high speed bus connection to a processingelement that contains a coordinator that makes decisions regarding theencoding system used with the incoming data. The coordinator may utilizea number of methods at its disposal and can offload the computation to afloating point calculator (which could be an external GPU). Thealgorithms may be updated periodically using an FPGA based controllerthat is able to adjust some of the algorithms or add new ones that makeuse of common resources, such as the associative network subsystem andBayesian subsystem.

As a further example, FIG. 12 is a diagram illustrating how a dataprocessing server or platform may be integrated with an embodiment ofthe inventive optimization methods (as implemented in an adaptor) inorder to provide a user with an improved experience when transferringrelatively large amounts of data over a network. The figure shows anarchitecture where an application or service is hosted on a server. Theapplication may be part of a data processing service (such as for abusiness function), a gaming experience, a simulation, a system model,etc. The server/platform communicates with an engine that processesincoming data into application data or user specific data (data that thesystem has learned from specific users). Note that the incoming datacould also be video, 3D mesh structures, etc. These are stored in theapplication database. When the data is sent to the client, it may useone or more of the optimization methods described herein (e.g., a sharedmeme table, Bayesian analysis engine, image reconstruction engine usingan associative neural network, space or shape curvature analysis, etc.).A corresponding element on the client side enables the data to bereconstructed if needed.

As described, the inventive adaptors enable devices and people tointeract with a virtual environment, system model, or simulation (andwith virtual entities) in an effective and efficient manner. These areneeded as viewing high fidelity content (such as a virtual realityenvironment) is currently unfeasible unless the content is downloadedahead of time. This restricts certain types of real-time behavior as itis not possible to update the content frequently enough for asatisfactory user experience. In some cases, such as a gamingenvironment, this may not be a serious problem, but in scenarios wherethe object(s) being viewed are updated continuously (such as asimulation, system model, or virtual landscape), this can pose seriouslimitations.

As described, the inventive adaptors provide a solution to this problemthrough implementation of one or more of several mechanisms andprocesses:

-   -   Logic that “predicts” (i.e., attempts to make an intelligent        prediction of) what the user or device is going to do next. One        method is to determine connections between events, such as        determining that when event A occurs there is a sufficiently        high likelihood of event B happening. This allows the system to        pre-compute the impact of event B on the state of a system or        environment.        -   For example, it is possible that an event could represent an            action such as moving in a 3D space. If the system notices            that moving to the left is followed by moving straight, then            it will precompute moving straight. This can be determined            using a Bayesian network. A key aspect to note is that as            everything is event driven, one can correlate events and            make event predictions, which translates to applying an            operator to an entity to cause it to move from one state to            another. As described, the use of a Bayesian network in the            context of the inventive system differs from the            conventional use of a Bayesian network, as it is possible            that a given event may occur multiple times in a graph. Note            this is more in accordance with human behavior and            experience.        -   With the approach described, it is possible to generate an            unsupervised learning machine (for example, the previously            described neural network) that is able to map out            dependencies and causal relationships (note that in a            typical Bayesian network, causality is implied but not            guaranteed). In the case of the inventive system, it is more            likely that the Bayesian network can identify a root cause            of an event, as it can analyze a sequence of possibly            repeating events and identify events that are highly likely            to cause a change in state. Another advantage of this            approach is that, because the network is unsupervised, it is            able to dynamically update itself as more information            arrives. In contrast, most implementations of Bayesian            networks have difficulty doing this (or are unable to)            because cyclical Bayesian networks are typically prohibited.        -   This approach is different from conventional predictive            systems that are used to optimize data traffic, as            conventionally, prediction is based on identifying data that            is most frequently accessed and storing that data on the            client side, with the less frequently accessed data stored            on the data server side (this is conventionally termed            “intelligent caching”).        -   The inventive approach “learns” from the user and makes            predictions about what the user is expected to do next, and            then prepares the configuration of the system for the            anticipated behavior. This is different from (and more            effective than) conventional methods because it is capable            of understanding more complex behavior than simply            identifying the most frequently used data.        -   For example, one implementation could be to break down            operations into logical “memes”, construct the system based            on instruction sequences and then make predictions as to            what will be executed next. In this approach, events            correspond to distinct operations. It is assumed that when            an event S is received, an action A(S) is executed, where A            is a logical function or operation, as represented by a            meme. By analyzing an example of the network, the system can            learn that if A happens, then it is likely that action B or            C will happen next (but not E or D). This allows the system            to pre-compute the actions of A, B, and C, on the state of            the system and store the results, or to pre-compute a            sequence of actions and pre-load an instruction sequence for            more rapid execution.        -   Note that this approach is different from conventional            approaches, such as data or instruction prefetching, because            in the case of pre-fetching a system is analyzing            application usage and then pre-loading data that the            application is expected to need. In contrast, in the case of            the inventive system, predictions are made at the            instruction level, followed by pre-loading and executing            those instructions, with the outcome being stored in memory.    -   Saliency based load balancing. Using this technique, the system        is able to understand when the user/device (actor) is about to        change its behavior and then notify the back-end processing        elements to prepare for the change in behavior (by, for example,        adding more computational resources, or pre-buffering data);        -   For example, an understanding of the relative saliency of an            event occurring within a system can be used to identify and            implement changes in resource usage needs, either by making            decisions as to when to de-allocate resources or            (conversely) when to allocate resources;        -   Based on the concept of saliency, it is also possible to            identify errors in an application, an emerging problem in a            flight control system, a potential health issue emerging in            a patient, impeding changes in the stock market or economic            system, or anomalous readings as a result of an intrusion in            an intrusion detection system that utilizes this approach to            indicate an unexpected or not previously experienced event            or situation. Saliency can also be used by a system to            identify possible errors, and to determine whether            variations in a user's or system's behavior are inconsistent            with previous behavior;    -   Event sequence caching. If the system detects a clustering of        events, then it will cache them and download content from the        back end server that is required for handling the set of events,        instead of all of the possibly relevant content;        -   An example of this use case is that of the operation of a            gaming system—in a gaming environment, if common things            occur (such as some area of virtual gaming being used), then            there could be markers in this gaming area. If the system            recognizes that a standard set of markers are encountered in            sequence or nearly in sequence as a result of a user taking            some action (such as opening a door and going down a            corridor), then when a user hits one of these markers, the            game system will have already downloaded the            resources/assets it needs to render the parts of the            environment that the user is likely to traverse next because            it recognizes that the marker was part of a sequence.            Similarly, the gaming system will not download other aspects            or areas of the game that are not expected to be used;        -   Another possible use case is that of a “smart” house that is            able to recognize that its owner typically follows a certain            sequence when entering the house—e.g., comes in, goes into            the living room, starts the fireplace, turns on the TV, goes            to the refrigerator, etc. In such as case, as soon as the            house detects that the owner is approaching the house, it            could start up the fireplace, turn on the TV, turn on the            lights in the living room etc.;    -   Data interpolation. In this situation, the system applies an        auto-associative neural network to reconstruct a set of data        using partial information (as was described with reference to        FIGS. 7-9);    -   Perception based logic that adapts to human perception (e.g.,        incorporation of high resolution in the center of vision, with a        more blurred exterior to mimic the behavior of the human eye and        mind); or    -   Application of a technique based on discrete differential        geometry—in this approach, rather than specifying every node in        a 3D mesh, the system can make reconstructions/interpolations        based on the anticipated curvature of a space or region. This        approach uses certain recognized local information about the        curvature/geometry of a shape to reduce the amount of data that        is needed to be transferred over a network to define that shape        to a client device or system.

A hardware based implementation of an embodiment of the inventiveadaptor will be responsible for a large amount of processing andtherefore will consume a significant amount of power. In someembodiments, the system may use one or more of several methods tominimize or reduce power consumption, such as:

-   -   Using FPGAs to handle most of the AI logic in hardware without        losing the ability to update the hardware;    -   Non-essential parts of the hardware are handled by a low power        processor;    -   Use a minimalist user interface;    -   No sensors or peripherals;    -   Aggressive sleep policies;    -   Passive liquid based cooling system;    -   Ultra-low power wireless components;    -   Low power buses for non-critical operations; or    -   Using the Peltier Effect for some amount of power recovery.        It is also possible to provide most, if not all, of the adaptor        as a software application, running on a standard computing        device such as a PC, tablet, cell-phone, IoT (internet of        things) device, printer, camera, or any other device with an        onboard computer. Naturally, the capabilities will be more        restricted as the adaptor will not utilize the same amount of        processing capability due to the resident device's power        constraints.

In some embodiments, the adaptor and/or a server side process mayutilize a central engine that understands or can discover certaincharacteristics of the type of data that is being transferred. In such acase, data that is entered into a server platform may be transformedinto a format that can be optimized. For example, programs/applicationsmay be analyzed probabilistically for likely instruction sets or logicflow, data may be broken down by analyzing content, etc. For example ifa photo is being sent, then the photo may be analyzed for faces, dogs,trees, elements of symmetry, translations, inversions, etc., and then areduced data set is prepared and sent to the client. Videos and moviescan similarly be broken down into scenes with only partial data beingsent from the platform to the client over a network, where it isreconstructed into the original scenes. These methods may produce theeffect of even higher degrees of data compression than conventionalmethods.

Unlike some conventional data compression methods, the inventiveapproaches are based on the structure of and relationships between theactual content that is being sent, with the optimization being based onthat content. In some embodiments, the inventive system can understandthat it is sending a picture of a face or a dog and then perform theoptimizations based on that knowledge (e.g., because it knows that theclient can recognize a dog or a person from incomplete information,based on previous experience, object symmetries, general relationshipsbetween elements, proportions, etc.).

Additional optimizations (such as those based on saliency) may permitmore efficient use of resources or the ability to pre-load data thatsupports the nominal behavior of a system. This can operate to make theexecution of an application more efficient when it is in the nominalmode. This approach is different from caching based on a pre-fetch of anapplication, as such a pre-fetch lacks a runtime understanding of thelogic flow of an application beyond the fact that the application isused frequently. In contrast, the inventive system can discover oridentify an application's nominal state and optimize resource allocationfor that expected behavior. If the application's nominal behaviorchanges, then the system can identify that change (via detection of asalient event) and adapt to make the new behavior the “nominal” one.

An example of this property is the following. For a given application,assume that User A uses function 1 but only if an hour has passed afterfunction 2 and function 3 have been called, and function 3 will only becalled if function 5 is called two hours before then. In such a case, iffunction 2 and function 5 are frequently called, then the system willoperate to load resources for function 1 and function 3 automatically atthe optimal times, as it will be able to determine that function 1 andfunction 3 are part of the application's nominal functioning.

In some embodiments, an integrated circuit (IC chip) may be used whichis capable of executing, or being used to execute, one or more of theinventive optimization processes, functions, or operations. In someembodiments, such a chip may be designed to be capable of being updatedto include changes to the set of algorithms being used to perform theoptimization processes. One way to implement this capability is to useprogrammable hardware, such as a combination of EEPROMs and FPGAs.

Note that as an optimization platform/system develops wider usage, thecentral association engine (as might be implemented in the form of oneor more coupled neural networks and/or modeling based on the conditionalprobabilities between events or program instructions) becomes more andmore powerful due to having a larger set of data to process and learnfrom. This can lead to order of magnitude improvements in managingnetwork traffic by being able to make advances in the area ofassociating/linking different pieces of data together that initiallyappear unrelated. This means, for example, that a client can reconstructcomplex data with less of the original data as the platform/systemcontinues to develop. It also means that the platform/system may becomecapable of effectively processing increasingly complex data sets as itcontinues to learn and develop its capabilities.

Although some of the description of one or more embodiments of theinventive system and methods have been in the context of theconstruction of gaming and simulations that include virtual environmentsand virtual entities, it should be noted that the described techniques,system architectures, and processes may be used to construct adistributed network of autonomous agents, each with certain basiccapabilities and functionality, and also with agent specific qualities.Further, each agent may belong to one or more sub-groups, sets, etc.,which do not include all of the agents.

In some cases, such agents may function as an operating entity in asimulated communications network, computing network, transportationnetwork, social network, environment, etc. Because of certainqualities/capabilities of the created agents/virtual entities, thoseentities are capable of moving between networks and establishingcommunications and participating in interactions such as monitoring,collecting data, etc.

The structure, operation, and administration of such agents and theirpossible implementations are discussed in greater detail in the UnitedStates Non-Provisional Patent Application entitled “System and Methodsfor Generating Interactive Virtual Environments” (Attorney Docket128683.0005), application Ser. No. 14/878,262, assigned to the sameentity as the present application and filed concurrently with thisapplication, and the contents of which are hereby incorporated byreference in their entirety for all purposes.

In accordance with at least one embodiment of the invention, the system,apparatus, methods, processes and/or operations for implementing one ormore types of data, event, or instruction optimizations may be wholly orpartially implemented in the form of a set of instructions executed byone or more programmed processors, such as a central processing unit(CPU) or microprocessor. Such processors may be incorporated in anapparatus, server, client or other computing device or platform operatedby, or in communication with, other components of the system.

As an example, FIG. 13 depicts aspects of elements that may be presentin a computer device and/or system 1300 configured to implement a methodand/or process in accordance with some embodiments of the presentinvention. The subsystems shown in FIG. 13 are interconnected via asystem bus 1302. Additional subsystems include a printer 1304, akeyboard 1306, a fixed disk 1308, and a monitor 1310, which is coupledto a display adapter 1312. Peripherals and input/output (I/O) devices,which couple to an I/O controller 1314, can be connected to the computersystem by any number of means known in the art, such as a serial port1316. For example, the serial port 1316 or an external interface 1318can be utilized to connect the computer device 1300 to further devicesand/or systems not shown in FIG. 13, including a wide area network suchas the Internet, a mouse input device, and/or a scanner. Theinterconnection via the system bus 1302 allows one or more processors1320 to communicate with each subsystem and to control the execution ofinstructions that may be stored in a system memory 1322 and/or the fixeddisk 1308, as well as the exchange of information between subsystems.The system memory 1322 and/or the fixed disk 1308 may embody a tangiblecomputer-readable medium.

In addition to the uses described herein, embodiments or applications ofthe inventive system and methods may be used for one or more of thefollowing purposes or goals:

-   -   Implementing a system that can develop a deeper understanding of        an environment by collecting data from a group of deployed        adaptors or other system elements that collect data in order to        identify associations between different data streams and uncover        relevant or associated patterns;    -   Creation of software applications and/or generating recommended        changes by collecting code snippets, probability, and popularity        maps to improve logic flow or the efficiency of user        interactions;    -   Generating predictions in complex systems and system models,        such as for the economy, climate, elections, the stock market,        large-scale transportation or communications networks, etc.;    -   Military/police systems—improving the recognition of people or        things based on minimal data, targeting, friend or foe        identification, etc.;    -   Archeology—generate complex reconstructions based on minimal        information regarding a location, building, etc.;    -   Behavioral prediction—assisting in understanding how people will        behave based on previous behavior and environmental states—may        be useful in predicting and/or preventing crime;    -   Reducing data storage requirements on the client and/or server        side while maintaining a desired level of content fidelity;    -   Based on behavioral information, likes/dislikes and the        associational engine that exists at the server level—creating        more effective social matching/dating applications, generating        product recommendations, implementing a more effective digital        personal assistant, developing highly customized applications        catering to individual uses or situations, etc.;    -   Creation of a more intelligent search engine based on the        collection of associational data that can improve relevancy of        search results:        -   Note that as the system learns from a user, the search data            can be made more contextual—for example when a frequent            shopper of fashion items executes a search using the word            “clutch”—the search should return small handbags. However,            an aficionado of cars should get search results relating to            car/transmission clutches;    -   Solving complex multivariate problems—the associational engine        can build a model of an environment (real or imagined) and        identify potentially related factors, events, etc. that        represent “insights”;    -   Effective fault or error prediction/detection using aspects of        the saliency properties of the system; and    -   Environmental or system intrusion detection (such as        unauthorized access to a building, system, database, etc.) using        aspects of the saliency properties of the system.

It should be understood that the present invention as described abovecan be implemented in the form of control logic using computer softwarein a modular or integrated manner. Based on the disclosure and teachingsprovided herein, a person of ordinary skill in the art will know andappreciate other ways and/or methods to implement the present inventionusing hardware and a combination of hardware and software.

Any of the software components, processes or functions described in thisapplication may be implemented as software code to be executed by aprocessor using any suitable computer language such as, for example,Java, Javascript, C++ or Perl using, for example, conventional orobject-oriented techniques. The software code may be stored as a seriesof instructions, or commands on a computer readable medium, such as arandom access memory (RAM), a read only memory (ROM), a magnetic mediumsuch as a hard-drive or a floppy disk, or an optical medium such as aCD-ROM. Any such computer readable medium may reside on or within asingle computational apparatus, and may be present on or withindifferent computational apparatuses within a system or network.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to the sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and/or were set forth in its entiretyherein.

The use of the terms “a” and “an” and “the” and similar referents in thespecification and in the following claims are to be construed to coverboth the singular and the plural, unless otherwise indicated herein orclearly contradicted by context. The terms “having,” “including,”“containing” and similar referents in the specification and in thefollowing claims are to be construed as open-ended terms (e.g., meaning“including, but not limited to,”) unless otherwise noted. Recitation ofranges of values herein are merely indented to serve as a shorthandmethod of referring individually to each separate value inclusivelyfalling within the range, unless otherwise indicated herein, and eachseparate value is incorporated into the specification as if it wereindividually recited herein. All methods described herein can beperformed in any suitable order unless otherwise indicated herein orclearly contradicted by context. The use of any and all examples, orexemplary language (e.g., “such as”) provided herein, is intended merelyto better illuminate embodiments of the invention and does not pose alimitation to the scope of the invention unless otherwise claimed. Nolanguage in the specification should be construed as indicating anynon-claimed element as essential to each embodiment of the presentinvention.

Different arrangements of the components depicted in the drawings ordescribed above, as well as components and steps not shown or describedare possible. Similarly, some features and sub-combinations are usefuland may be employed without reference to other features andsub-combinations. Embodiments of the invention have been described forillustrative and not restrictive purposes, and alternative embodimentswill become apparent to readers of this patent. Accordingly, the presentinvention is not limited to the embodiments described above or depictedin the drawings, and various embodiments and modifications can be madewithout departing from the scope of the claims below.

The terms and expressions which have been employed in the foregoingspecification are used therein as terms of description and not oflimitation, and there is no intention in the use of such terms andexpressions of excluding equivalents of the features shown and describedor portions thereof, it being recognized that the scope of the inventionis defined and limited only by the claims which follow.

What is claimed is:
 1. A process to reduce the latency experienced by aclient device interacting with a server device, where the client andserver devices are part of a common system, comprising: for a pluralityof pairs of events occurring within the common system, determining aprobability of occurrence between a first event of a pair and a secondevent of the pair; constructing a representation of a network, thenetwork including multiple nodes with each node representing an eventoccurring within the common system, wherein a transition between twonodes in the network is associated with the probability of occurrencebetween the event represented by a first node and the event representedby the second node; for each of a plurality of possible events occurringwithin the common system and based on a state of the common system andthe constructed representation, determining one or more likely states ofthe common system that may result in response to each of the pluralityof possible events; providing each of the one or more likely states ofthe common system that may result in response to each of the pluralityof possible events to the client device; operating the client device togenerate one of the plurality of possible events; and based on thegenerated event, configuring the common system to be in the stateresulting from the response to the generated event.
 2. The process ofclaim 1, wherein the pairs of events are associated with the executionof an instruction or set of instructions by an electronic processingelement.
 3. The process of claim 2, wherein execution of the instructionor instruction set results in the operation of a virtual environment,simulation, or the processing of an image or video.
 4. The process ofclaim 1, wherein the server device is a data processing platform thathosts one or more data processing applications and the client device andserver device are communicatively coupled by a network.
 5. The processof claim 4, wherein the hosted data processing applications include oneor more of an image processing application, a video processingapplication, or an application for processing business related data. 6.The process of claim 1, further comprising: performing an optimizationprocess that includes an associative neural network, the neural networkincluding elements operable to determine that a first event and a secondevent are generated within a specified time interval; using theassociative neural network to identify an association between a firstset of data elements and a second set of data elements; and providingthe first set of data elements to the client device.
 7. The process ofclaim 6, further comprising reconstructing the second set of dataelements based on the first set of data elements without being providedthe second set of data elements by the server device.
 8. The process ofclaim 7, wherein the first set of data elements are a portion of animage, and the combination of the first set of data elements and thesecond set of data elements form the image.
 9. A process for reducing anamount of data that needs to be transferred over a network, comprising:performing an optimization process that includes an associative neuralnetwork, the neural network including elements operable to determinethat a first event and a second event are generated within a specifiedtime interval; using the associative neural network to identify anassociation between a first set of data elements and a second set ofdata elements; and providing the first set of data elements to thenetwork.
 10. The process of claim 9, further comprising reconstructingthe second set of data elements based on the first set of data elementswithout being provided the second set of data elements over the network.11. The process of claim 9, wherein the first set of data elements are aportion of an image, and the combination of the first set of dataelements and the second set of data elements form the image.
 12. Theprocess of claim 9, further comprising performing a second optimizationprocess to determine one or more executable instructions that are partof a data processing application and that may be executed by a clientdevice coupled to the network.
 13. The process of claim 12, wherein thesecond optimization process further comprises: determining a logic flowfor the data processing application by, for each of a plurality of afirst executable instruction or set of executable instructions and asecond executable instruction or set of executable instructions,determining a probability of occurrence between the first executableinstruction or first set of executable instructions and the secondexecutable instruction or second set of executable instructions;constructing a representation of the logic flow of the data processingapplication, the representation including multiple nodes, wherein atransition between two nodes is associated with the probability ofoccurrence between the first node and the second node; based on acurrent or expected executable instruction or instructions and theconstructed representation, determining one or more instructions thatmay be executed in response to the current or expected executableinstruction or instructions; and providing the one or more instructionsthat may be executed in response to the current or expected executableinstruction or instructions to the client device.
 14. The process ofclaim 13, wherein the data processing application is one of an imageprocessing application, a video processing application, or anapplication for processing business related data.